<html>
<head>
	<title>METviewer Documentation</title>
	<link rel="stylesheet" type="text/css" href="mv_doc.css" />
	<link rel="shortcut icon" href="include/ral_icon.ico" type="image/x-icon"/>
</head>
<body>

<p class="loc" style="padding-top:10px">
<b>Location:</b> <a class="loc" href="index.html">Home</a> &#187; Database Loading Module
</p><hr/>

<h2>METviewer Documentation - Database Loading Module</h2>

<p>The database loading module is used to insert, update and delete MET output data in the database.  The tool is invoked using the mv_load script.  The usage statement:</p>

<p class="term">
----  MVLoad  ----<br/><br/>
Usage:  mv_load<br/>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;load_spec_file<br/>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[-index]<br/><br/>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;where   "load_spec_file" specifies the XML load specification document<br/>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"-index" indicates that no data should be loaded, and only the indexing commands applied<br/><br/>
----  MVLoad Done  ----<br/>
</p>

<p>
The <span class="term">load_spec_file</span> passed to the loading module  information about the MET output files to load into the database.  It is an XML file whose
top-level tag is <span class="code">&lt;load_spec&gt;</load_spec></span> which contains the following elements, divided into functional sections:

<ul>
	<li><span class="code">&lt;connection&gt;</span> see <a href="common_xml.html">common xml</a></li><br/>
	
	<li><span class="code">&lt;date_list&gt;</span> see <a href="common_xml.html">common xml</a></li><br/>
	
	<li><span class="code">&lt;load_stat&gt;</span>: <span class="code">true</span> or <span class="code">false</span>, indicating whether or not to load stat data</li>
	<li><span class="code">&lt;load_mode&gt;</span>: <span class="code">true</span> or <span class="code">false</span>, indicating whether or not to load mode data</li>
	<li><span class="code">&lt;load_mtd&gt;</span>: <span class="code">true</span> or <span
			class="code">false</span>, indicating whether or not to load MODE TD data</li>
	<li><span class="code">&lt;load_mpr&gt;</span>: <span class="code">true</span> or <span class="code">false</span>, indicating whether or not to load matched pair data</li>
	<li><span class="code">&lt;load_orank&gt;</span>: <span class="code">true</span> or <span class="code">false</span>, indicating whether or not to load observed rank data</li>
	<li><span class="code">&lt;force_dup_file&gt;</span>: <span class="code">true</span> or <span class="code">false</span>, indicating whether or not to force loading of path/files 
		that are already present</li>
	<li><span class="code">&lt;verbose&gt;</span>: <span class="code">true</span> or <span class="code">false</span>, depending on the desired volume of output from the load module</li>
	<li><span class="code">&lt;insert_size&gt;</span>: integer indicating the number of MET output file rows that are inserted with each INSERT statement</li>
    <li><span class="code">&lt;stat_header_db_check&gt;</span>: <span class="code">true</span> or <span class="code">false</span>, indicating whether a database query check for stat header
    		information should be performed - WARNING: enabling this feature could significantly increase load time</li>
	<li><b style="color:#666666">NOTE:</b> <span class="code">&lt;stat_header_table_check&gt;</span> and has been removed; remove it from your XML load specification document</li>
	<li><span class="code">&lt;mode_header_db_check&gt;</span>: <span class="code">true</span> or <span class="code">false</span>, indicating whether a database query check for header
		information should be performed - WARNING: enabling this feature could significantly increase load time</li>
	<li><span class="code">&lt;mtd_header_db_check&gt;</span>: <span class="code">true</span> or
		<span class="code">false</span>, indicating whether a database query check for header
			information should be performed - WARNING: enabling this feature could significantly increase load time</li>
	<li><span class="code">&lt;drop_indexes&gt;</span>: <span class="code">true</span> or <span class="code">false</span>, indicating whether database indexes should be dropped
		prior to loading new data</li>
	<li><span class="code">&lt;load_indexes&gt;</span>: <span class="code">true</span> or <span class="code">false</span>, indicating whether database indexes should be created
		after loading new data</li>
	<li><span class="code">&lt;group&gt;</span>: The name of the group for the user interface</li>
	<li><span class="code">&lt;description&gt;</span>: The description of the database.</li><br/>

	<li>
		<span class="code">&lt;load_files&gt;</span>: a list structure containing individual MET output files to load into the database
		<ul>
			<li><span class="code">&lt;file&gt;</span>: contains a single MET output file to load</li>
		</ul>		
	</li><br/>

	<li><span class="code">&lt;folder_tmpl&gt;</span>: a template string describing the file structure of the input MET files, which is populated with values specified in the
		<span class="code">&lt;load_val&gt;</span> tag structure</li>
	<li>
		<span class="code">&lt;load_val&gt;</span>: a tree structure containing values used to populate the <span class="code">&lt;folder_tmpl&gt;</span> template
		<ul>
			<li><span class="code">&lt;field&gt;</span>: a template value, whose name is specified by the attribute name, and whose values are specified by it's chilren 
				<span class="code">&lt;val&gt;</span> tags</li>
			<ul>
				<li><span class="code">&lt;val&gt;</span>: a single template value which will slot into the template in the value specified by the parent field's name</li>
				<li><span class="code">&lt;date_list&gt;</span>: specifies a previously declared <span class="code">&lt;date_list&gt;</span> element, using the name attribute,
					which represents a list of dates in a particular format</li>
			</ul>
		</ul>		
	</li><br/>

	<li>
		<span class="code">&lt;line_type&gt;</span>: a list structure containing the MET output file line types to load.  If omitted, all line types are loaded.
		<ul>
			<li><span class="code">&lt;val&gt;</span>: contains a single MET output file line type to be loaded, for example CNT, CTS, SL1L2 or NBRCTC</li>
		</ul>		
	</li><br/>

	<li><span class="code">&lt;load_note&gt;</span>: if present, creates a record in the instance_info database table with a note containing the body of this tag</li>
	<li><span class="code">&lt;load_xml&gt;</span>: <span class="code">true</span> or <span class="code">false</span>, indicating whether or not to save the load xml;
		only effective if <span class="code">&lt;load_note&gt;</span> is present - default: <span class="code">true</span></li>

</ul>
</p>

<p>
Here is a simple example:
</p>

<p class="code">

&lt;load_spec&gt;<br/>
&nbsp;&nbsp;&lt;connection&gt;<br/>
&nbsp;&nbsp;&nbsp;&nbsp;&lt;host&gt;<span class="term">kemosabe:3306</span>&lt;/host&gt;<br/>
&nbsp;&nbsp;&nbsp;&nbsp;&lt;database&gt;<span class="term">metvdb_hwt</span>&lt;/database&gt;<br/>
&nbsp;&nbsp;&nbsp;&nbsp;&lt;user&gt;<span class="term">pgoldenb</span>&lt;/user&gt;<br/>
&nbsp;&nbsp;&nbsp;&nbsp;&lt;password&gt;<span class="term">pgoldenb</span>&lt;/password&gt;<br/>
&nbsp;&nbsp;&lt;/connection&gt;<br/><br/>
&nbsp;&nbsp;&lt;date_list name="<span class="term">folder_dates</span>"&gt;<br/>
&nbsp;&nbsp;&nbsp;&nbsp;&lt;start&gt;<span class="term">2010051914V</span>&lt;/start&gt;<br/>
&nbsp;&nbsp;&nbsp;&nbsp;&lt;end&gt;&lt;date_offset&gt;&lt;day_offset&gt;<span class="term">0</span>&lt;/day_offset&gt;&lt;hour&gt;<span class="term">6</span>&lt;/hour&gt;&lt;/date_offset&gt;&lt;/end&gt;<br/>
&nbsp;&nbsp;&nbsp;&nbsp;&lt;inc&gt;<span class="term">3600</span>&lt;/inc&gt;<br/>
&nbsp;&nbsp;&nbsp;&nbsp;&lt;format&gt;<span class="term">yyyyMMddHH'V'</span>&lt;/format&gt;<br/>
&nbsp;&nbsp;&lt;/date_list&gt;<br/><br/>
&nbsp;&nbsp;&lt;verbose&gt;<span class="term">false</span>&lt;/verbose&gt;<br/>
&nbsp;&nbsp;&lt;insert_size&gt;<span class="term">1</span>&lt;/insert_size&gt;<br/>
&nbsp;&nbsp;&lt;mode_header_db_check&gt;<span class="term">true</span>&lt;/mode_header_db_check&gt;<br/>
&nbsp;&nbsp;&lt;drop_indexes&gt;<span class="term">false</span>&lt;/drop_indexes&gt;<br/>
&nbsp;&nbsp;&lt;apply_indexes&gt;<span class="term">true</span>&lt;/apply_indexes&gt;<br/>	
&nbsp;&nbsp;&lt;group&gt;<span class="term">Group name</span>&lt;/group&gt;<br/>
&nbsp;&nbsp;&lt;load_stat&gt;<span class="term">true</span>&lt;/load_stat&gt;<br/>
&nbsp;&nbsp;&lt;load_mode&gt;<span class="term">true</span>&lt;/load_mode&gt;<br/>	
&nbsp;&nbsp;&lt;load_mtd&gt;<span class="term">true</span>&lt;/load_mtd&gt;<br/>
&nbsp;&nbsp;&lt;load_mpr&gt;<span class="term">false</span>&lt;/load_mpr&gt;<br/><br/>
&nbsp;&nbsp;&lt;folder_tmpl&gt;<span class="term">/d1/data/{model}/{vx_mask}/{valid_time}</span>&lt;/folder_tmpl&gt;<br/>
&nbsp;&nbsp;&lt;load_val&gt;<br/>
&nbsp;&nbsp;&nbsp;&nbsp;&lt;field name="<span class="term">model</span>"&gt;<br/>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;val&gt;<span class="term">arw</span>&lt;/val&gt;<br/>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;val&gt;<span class="term">nmm</span>&lt;/val&gt;<br/>
&nbsp;&nbsp;&nbsp;&nbsp;&lt;/field&gt;<br/><br/>		
&nbsp;&nbsp;&nbsp;&nbsp;&lt;field name="<span class="term">valid_time</span>"&gt;<br/>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;date_list name="<span class="term">folder_dates</span>"/&gt;<br/>
&nbsp;&nbsp;&nbsp;&nbsp;&lt;/field&gt;<br/><br/>		
&nbsp;&nbsp;&nbsp;&nbsp;&lt;field name="<span class="term">vx_mask</span>"&gt;<br/>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;val&gt;<span class="term">FULL</span>&lt;/val&gt;<br/>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;val&gt;<span class="term">SWC</span>&lt;/val&gt;<br/>
&nbsp;&nbsp;&nbsp;&nbsp;&lt;/field&gt;<br/>
&nbsp;&nbsp;&lt;/load_val&gt;<br/>
&lt;/load_spec&gt;<br/>
</p>

<p>
In this example, the load module would attempt to load any files with the suffix .stat in the following folders.  This list would end at the date specified by the
<span class="code">&lt;date_offset&gt;</span> named <span class="term">folder_dates</span>.   
</p>

<p class="term" style="padding-left: 40px">
/d1/data/arw/FULL/2010051914V<br/>
/d1/data/arw/SWC/2010051914V<br/>
/d1/data/nmm/FULL/2010051914V<br/>
/d1/data/nmm/SWC/2010051914V<br/>
/d1/data/arw/FULL/2010051915V<br/>
/d1/data/arw/SWC/2010051915V<br/>
/d1/data/nmm/FULL/2010051915V<br/>
/d1/data/nmm/SWC/2010051915V<br/>
...<br/>
</p><br/>

<h3>Troubleshooting</h3>

<p style="padding-left:30px; padding-right:30px">
<table cellpadding="3" cellspacing="0" border="1">
	<tr>
		<td class="header" align="right">Error:</td>
		<td><b>
		**  ERROR: Caught class com.mysql.jdbc.exceptions.MySQLIntegrityConstraintViolationException: Duplicate entry 
		'CT07-NMM-LIN-R2-0-2005-07-15 12:00:00-2005-07-15 12:00:00-0-2005' for key 2
		</b></td>
	</tr>
	<tr>
		<td class="header" align="right">Solution:</td>
		<td style="font-size:10px">
		This error is caused by trying to insert a stat_header record into the database when an identical one already exists.  If identical stat_header information
		is present in more than one stat file, set the <span class="code">&lt;stat_header_db_check&gt;</span> value to <span class="term">true</span>.  This setting
		will reduce performance, because the stat_header table is checked for duplicate stat_header each time a row is inserted.  However, if a stat_header row
		already exists in the table with the insert information, then the existing record will be used instead of trying to insert a dupilcate.
		</td>
	</tr>
</table>
</p>

<br/><br/>

</body>
</html>
